• Adds a new column family for cached cross chunks edges.
• Adds MaxAgeGCRule for previous column family with supervoxel cross chunk edges; only needed during ingest and they get deleted eventually.
• Edits make use of cached cross chunk edges.

Summary of changes in pychunkedgraph.ingest:

• Layer 2 creation is mostly unchanged; stores cross chunk edges with supervoxels
  • The column family used to store these edges now has a max age garbage collection rule
  • During ingest, these edges can be used to cache higher layer cross chunk edges; will be deleted eventually by BigTable's garbage collection routines.
• When ingesting layer 3, cross edges for children (layer 2) get updated and "lifted" by using the previously mentioned supervoxel cross chunk edges, these have a different column family so they're retained forever.
  • At the same time, cross edges for parents at layer 3 will get created by merging cross edges of their children, these are intermediate and will be lifted when ingesting the next parent layer.
• For each layer > 3 until root layer:
  • Update children cross chunk edges by "lifting" the edges created during the previous layer ingest.
  • Add parent cross chunk edges by merging children cross chunk edges; they will be updated when ingesting the next layer.

This assumes all chunks at lower layer have been created before creating the current layer so we can no longer queue parent chunk jobs automatically when its children chunks are complete.

We must now ingest/create one layer at a time.

Summary of changes in pychunkedgraph.graph.edits:

• Edits are expected to be faster now; going to layer 2 to extract cross chunk edges is no longer necessary since they're cached at each layer.
• During an edit, these cached cross chunk edges must be updated from both directions - to and from the newly created nodes and its existing neighbors.
  • Most changes in this module are to handle this step.
  • Caching these edges has also made the edits logic simpler and cleaner.
  • When updating new cross edges, we need to ensure descendants get replaced by the highest parent.
  • For splits, we need to filter out inactive cross edges after the local graph is read from bucket storage.